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Results from the survey of faculty who teach in an alternative teacher 
certification program provided a framework to develop a holistic 
programmatic assessment of student learning outcomes for a newly 
developed online program. Research highlights the process of determining 
student learning outcomes identification and appropriate courses and best 
practices for online assessments to provide for program quality, formative 
and summative assessments, and data for accreditation purposes. 


To be or not to be...online? This 
was a question pondered by an 
alternative teacher certification program 
at a regional university in northeast 
Texas with a reputation of producing 
high quality educators for the state of 
Texas with 738 educators certified in 
2006, third in university production of 
teachers and sixth out of 135 
certification entities. Competition 
towards preparing future educators 
comes from a variety of online programs 
ranging from the for-profit, private, and 
public colleges and universities. Online 
Alternative Certification programs’ 
enrollment is on the rise; however, there 
is a lack of evidence that these programs 
are providing quality teacher education. 
The convenience online education meets 
the needs of today’s adult student; 
therefore, the answer to the question 
became clear: to be online. The 
challenge then was to not sacrifice the 
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known quality that existed within the 
teacher preparation programs within 
higher education. 

THEORETICAL FRAMEWORK 

Faced with accountability demands 
universities must develop broader 
assessment practices to document 
evidence of student learning outcomes; 
coupled with the increased demand for 
more online courses, assessment 
presents a unique challenge. Angelo 
(May, 1999) reminds us that assessment 
is first to improve student learning, and 
second, an accountability measure of the 
quality of student learning. The study is 
guided by two theoretical frameworks; 
first, that quality educational programs 
should include student involvement, 
high expectations, assessment and 
feedback (Astin as cited in Skawinski & 
Thibodeau, 2002), and second, the nine 
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principles of good practice for assessing 
student learning (Astin, Banta, Cross, El- 
Khawas, Ewell, Hutchings, et al, 1996). 
Online learning provides students with 
educational opportunities in a different 
environment; however, the tenants of 
sound educational practice remain the 
same. Just as in traditional university 
programs, online programs are held 
responsible for and expected to provide 
evidence of the process, growth and 
change in student learning over time. 
The identification, documentation of 
evidence and assessment of student 
learning outcomes are just one example 
of a critical issue faced by faculty and 
administration when making the 
transition to online. 

Evidence of student learning, not 
only through grades from traditional 
assessment practices, is an expected and 
increasing demand by the various 
educational stakeholders who want 
documentation that demonstrates the 
entire process of learning for 
accountability (Heaney, 1990; Ruhland 
& Brewer, 2001;Terheggen, Prabhu & 
Lubinescu, 2000; Villano, 2005). 
Multiple assessments should be used for 
diagnostic, process and outcome 
purposes to demonstrate effective 
learning (Padro & Hurley, 2005). These 
learning outcomes assessment methods 
may vary depending on the academic 
program goals, the philosophical beliefs 
of assessment practices of the faculty, 
college and/or department, and measures 
whether students have reached the 
learning expectations (Beghetto, 2004; 
Macdonald, 2002). 

The increased demand for 
university programs and courses to be 
provided online presents an additional 
challenge. Progress is often limited in 
offering new distance education courses 
and programs when there is not a 
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conceptual understanding that is 
consistent with programmatic and 
institutional values and goals (Garrison, 
1989). Often instructors question how 
they will assess their students’ learning 
online as they become aware that the 
more traditional methods do not apply 
(Runyon, 2005), and do not want to 
change their instructional style for fear 
that online instruction will not be as 
effective as their current teaching style 
(Levy, 2004). Therefore, with 
appropriate education new ways of 
thinking about teaching and learning can 
replace faculty concerns about online 
teaching and student assessment (Bates, 
2000; Burgess, 1994). 

Assessment must be a continuous, 
ongoing process to provide constant 
monitoring of student progress to 
improve both learning and teaching 
(Fenno, 2003). Data from the assessment 
of student learning outcomes can be 
utilized by faculty and department chairs 
to improve upon the program’s quality 
through revision of course materials, 
learning practices, teaching methods and 
assessment measures. 

PROBLEM AND PURPOSE 

While teaching more online 
courses, university’s’ challenge is to 
develop broader assessment practices to 
document and provide evidence of 
student learning. The problem for one 
Alternative Teacher Certification 
Program (ATC Program) in northeast 
Texas was not to decide whether to be 
online, but how to develop a process that 
implements a quality online program 
which effectively monitors student 
learning over time via learning outcomes 
assessment, and provides feedback for 
continuous improvement. The purpose 
of this exploratory case study was to 
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identify from the instructors within the 
ATC program learning outcomes they 
perceived as important and the best 
methods in which to assess them. 
Additionally, it was important to 
determine how feedback should be 
provided to inform practice. 

METHODS AND PROCEDURES 

This exploratory case study 
consisted of surveys disseminated during 
the ATC Program online development 
meeting in the 2006 fall semester to five 
full-time faculty teaching one or more 
courses within the program. All 
participants were provided with a 
consent form to inform them of the study 
and ask for their participation. Faculty 
who participated in the study taught 
courses in the Departments of 
Curriculum and Instruction (secondary, 
8-12, middle school 4-8, and elementary, 
preK-4), and Special Education and 
Psychology in the College of Education 
and Fluman Services. 

The survey consisted of six open- 
ended questions and three ranking 
questions based upon the theoretical 
framework of what constitutes a quality 
educational program and principles of 
student learning outcomes assessment. 
The survey was pilot tested and revised 
prior to dissemination. To provide 
anonymous participation, subjects were 
asked to complete the survey and return 
it to the researchers via campus mail 
(Appendix A). 

Because the ATC Program was 
designing and developing courses for 
online/partial online delivery, the 
researchers utilized this opportunity to 
obtain feedback from the instructors. 
Participants were asked to identify 
important learning outcomes for the 
student to achieve upon completion of 
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the program. Participants were also 
asked to rank assessment methods from 
a list ranging from traditional to 
alternative. A learning outcome was 
defined as the measurement of how a 
student’s university experience 
supported their development as 
individuals and described the 
knowledge, skills, abilities and attitudes 
one is able to demonstrate upon 
completion of a program. A learning 
outcome is not what the instructor does 
to the student, but rather what the 
instructor expects the student to know 
and be able to do as a result of the 
teaching and learning experience. How 
one measures learning is part of one’s 
teaching philosophy; therefore, 
participants were also asked to describe 
their philosophy on assessment. 

The researchers were interested in 
knowing what continuous improvement 
practices were currently used for 
program improvement in addition to 
participant perceptions on ideal methods 
for obtaining feedback to facilitate 
continuous improvement. The final two 
survey questions addressed faculty 
perceptions on the ability to assess 
learning outcomes in the online 
environment and faculty comfort level in 
teaching in this medium. 

Data were examined and reported 
focusing on the four topics within the 
survey: program student learning 

outcomes, faculty philosophy and 
methods of assessment, continuous 
improvement, and assessing student 
learning outcomes online. Responses 
from the open-ended questions were 
analyzed utilizing the constant- 
comparative method (Glaser & Strauss, 
1967) to identify common themes/ideas. 
Each theme/idea/unit was color coded in 
order to provide a visual representation 
and allow the researcher to isolate and 
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categorize information from the open- 
ended responses. Responses to the three 
ranking questions for assessment 
methods tied to learning outcomes were 
compiled and reported. Results from the 
data analysis are presented by survey 
topic. 

RESULTS AND DISCUSSION 

The results from this study focused 
upon the first step in formulating a 
process for developing program learning 
outcomes. These results may lead to 
determining the best assessment 
practices for the online environment, as 
well as how to best utilize assessment 
results for continuous programmatic 
improvement. 

Program Student Learning 
Outcomes 

Step one in our process was to 
identify student learning outcomes at the 
program level by asking faculty what a 
student should know upon completion of 
the ATC Program. The importance 
behind this step is to develop a shared 
vision among faculty as this is critical 
when multiple sections of the same 
course are taught and/or if adjuncts teach 
within the program. Participants were 
asked whether or not the identification of 
student learning outcomes at the 
program level was important and what 
emphasis they placed upon the 
identification of these learning 
outcomes. All participants agreed that it 
was highly important to identify learning 
outcomes for the ATC program with the 
exception of one individual believing it 
to be somewhat important. These results 
were not surprising. What was 
interesting were not the outcomes that 
were identified, but rather the manner in 
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which these learning outcomes were 
written: as broad, unclear, 

generalizations. This indicates the need 
to further explore the use of learning 
outcomes and the purposes they serve. 
Often the tendency of faculty is to thi nk 
within the context of a course and not 
the impact upon the total program. 
While it is natural to identify outcomes 
established within our courses, it is vital 
to ensure that everyone, students 
included, knows and understands the 
program’s goals and objectives in order 
to recognize how a course so seemingly 
different from another does fit into the 
overall program goals. 

After an analysis of participants’ 
responses to the ATC Program survey 
question on what they believe to be the 
most important student learning 
outcomes, the following categories 
emerged: Classroom and Discipline 
Management, Teaching, Learning and 
Assessment Strategies, Culture and 
Diversity, Effective Communication, 
Law and Ethics, and Technology 
Integration. These categories then 
become the program outcomes for which 
best methods of assessment are 
identified later in the process. 

In the classroom management and 
discipline category, participants 
identified that it is important for students 
to know, understand, and implement 
effective strategies prior to completing 
the program. In the teaching, learning 
and assessment strategies category, 
participants indicated students should be 
able to design and develop “engaging, 
creative, integrated lessons” that 
incorporate higher-order thinking to 
reach different ability levels and learning 
styles. For the culture and diversity and 
effective communication categories, 
students within the program should be 
able to know and understand 
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characteristics of their future students, 
factors that impact their future students’ 
learning, and have the ability to work 
with and communicate with parents of 
students. One participant took this a step 
further and expressed the importance of 
the student within the ATC Program 
having the ability to not only work with 
their future colleagues and the 
community, but have knowledge of the 
legal and ethical requirements creating a 
law and ethics category. The final 
category, technology integration, 
expressed the need to prepare future 
teachers to utilize technology effectively 
and efficiently in the classroom. 

Faculty Philosophy & Methods of 
Assessment 

Common across all responses was 
that assessment should be authentic, 
ongoing and meaningful to the learner. 
Participants believed that assessment 
should provide multiple opportunities for 
the student to demonstrate what they 
know, align to the course objectives, and 
align with the learning style of the 
individual. Assessment methods should 
be conducted both formally and 
informally, utilizing a variety of methods 
such as oral feedback, written 
assignments and tests, and be both 
formative and summative; however, 
assessment in terms of the traditional 
measures, while necessary, was seen as 
often overused. Assessment should push 
beyond traditional practice, and provide 
the learner with real world assessments 
or scenarious in which they are likely to 
face in one of their classrooms. For 
example, students were asked to role 
play situations requiring “ethical 
decisions over matters that are not clear 
cut.” 
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A personal philosophy is just that, 
personal. Therefore, it is not unexpected 
that the assessment philosophies 
reflected were unique and varied, but 
they all indicate the participants’ 
commitment to provide their students 
with multiple opportunities and rigorous 
practice in order to be successful in their 
future careers. 

Participants were asked to 
prioritize the best methods of assessment 
for the program learning outcomes 
identified. Methods ranged from 
traditional assessments such as paper 
and pencil tests and program 
comprehensive exams to more authentic 
and innovative ones like program 
portfolio assessment, presentations, 
personal interviews, and peer/self 
evaluation. 

For the classroom and discipline 
management category, capstone/field 
experience, peer/self evaluation, 
projects, personal interviews and 
presentations were ranked high by all 
participants over paper/pencil tests, 
program portfolio assessment and 
program oral exams. Methods of 
assessment for the teaching, learning and 
assessment strategies category were 
again given high priority in the areas of 
presentation, capstone/field experience, 
and program portfolio assessment. 
Similar to the classroom and discipline 
management category, paper/pencil tests 
and program oral exams were not seen 
as an effective method of assessment. 

The culture and diversity category 
identified capstone/field experience as a 
top method of assessment, but the 
second top method differed drastically 
among participants. For example, 
paper/pencil tests ranked high by one 
individual and as a last resort by another 
who selected personal interviews as a 
second method of assessment. 
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Methods to assess learning 
outcomes within the effective 
communication category included 
capstone/field experiences, personal 
interviews, peer/self evaluation, projects 
and presentations. Like the classroom 
and discipline management and 
teaching, learning and assessment 
strategies categories, paper/pencil tests 
and program oral and written 
comprehensive exams were not selected. 

Methods of assessment for the 
technological integration category had 
program portfolio assessment, 
presentations and projects as the top 
means in which to measure students’ 
performance as opposed to the more 
traditional measures. And finally, the 
methods for assessing the law and ethics 
category favored program portfolio 
assessment, projects and presentations 
over traditional practices. 

Results from the best methods for 
assessment of learning outcomes 
identified correspond to the 
philosophical beliefs of the participants. 
This indicates their willingness to move 
beyond traditional assessment practices 
in order to provide their students with 
the authentic learning experiences they 
will surely need once they are in the 
classroom. One must ask the question, if 
authentic and meaningful measurement 
of students’ knowledge, skills and 
abilities is seen as a priority of the 
faculty that prepare these future teachers, 
why does becoming a teacher hinge 
upon passing a standardized test? Why 
not the rich evidence of what these 
students are able to do within the 
program? 

Continuous Improvement 

Continuous improvement of 
program quality can only be enhanced 
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with the results from program student 
learning outcomes which assists in 
providing evidence of the effectiveness 
of the teaching and learning experiences 
of the student. So often, finals are taken, 
course evaluations conducted, grades 
distributed, and the semester comes to a 
close. For continuous improvement we 
should look to our students, courses and 
program learning outcomes to determine 
to what degree these learning outcomes 
were accomplished as indicators of the 
success rate of our students. Continuous 
improvement is vital to ensure the 
quality of our program and our faculty, 
and to remain in touch with the demands 
of the society we and our students 
ultimately serve. 

Faculty were asked what was 
currently being done to provide feedback 
and how they believed feedback should 
be provided for continuous improvement 
of the ATC Program. Other than end of 
the semester student summative course 
evaluations, which are not program 
specific, and individual efforts to collect 
information with formative assessments, 
final projects, and portfolio evaluations, 
there is no formal attempt for providing 
feedback on program learning outcomes. 
For individual self-designed formative 
assessments, the information/feedback 
works well, but there is no effort to 
make improvements beyond his/her 
personal development. 

Suggestions for providing feedback 
for continuous improvement included (a) 
increased communication among 
instructors of all courses in the ATC 
Program on a regular basis to align 
learner outcomes, and (b) to obtain 
feedback from administrators and 
mentors to assess students’ strengths and 
weaknesses. While the sample size for 
this case study is extremely small, the 
results are not unexpected and most 
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likely reflect the current common 
practices of many. More importantly, 
they do indicate the need to adopt a more 
inclusive continuous improvement 
strategy. 

Assessing Student Learning 
Outcomes Online 

Moving courses and a program to 
an online environment raises many 
concerns, issues and fears, all of which 
are valid and must be acknowledged. A 
primary question/concern centers on 
assessment and the ability to do so 
effectively and fairly. 

Participants were asked what their 
thoughts were in regards to the ability to 
assess student learning outcomes in an 
online course. One participant, an 
experienced online teacher, believed that 
outcomes can be assessed effectively, 
but only when the numbers within the 
courses are manageable and the 
instructor has an understanding of 
assessment methods. Another individual 
expressed that the multiple assessments, 
such as course projects, she intended to 
use should effectively provide her with 
evidence of students’ achievement of 
learning outcomes. Participants not yet 
comfortable teaching in the online 
environment reflected a common 
concern which was the lack of the 
physical presence of the student leading 
to interpreting meaning of words on 
paper as opposed to seeing and hearing 
student-student and student-teacher 
interactions in the traditional classroom. 
Another noted how complex the 
teaching and learning process can be and 
in turn making assessment equally as 
complex and stated: 

The nuances of teaching teachers 

often takes the shape of a dynamic 
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and wide ranging dialogue as 
students and instructor explore the 
complex facets of teaching. My 
concern is that these dialogues and 
the learning that occurs during 
them will be different for online 
instruction. (Survey Participant, 
2006) 

CONCLUSIONS AND 
RECOMMENDATIONS 

The preliminary findings from this 
case study have helped to lay the 
foundation for further development and 
refinement of a process (Figure 1) that in 
the end will provide faculty and students 
with a clear direction as to the 
expectations for learning outcomes, 
assessment and continuous 

improvement. This paper focused upon 
the first step in a five step process which 
was the identification of program 
learning outcomes and the courses in 
which they belong. As a result, the 
program faculty will have the 
opportunity to engage in a dialogue 
about the expected learning outcomes of 
a graduate of the ATC Program leading 
to the development of a common 
language among categories/learning 
outcomes, step two. 

Another advantage of creating a 
common language for programmatic 
learning outcomes will be the ability to 
inform students upon entrance to the 
program, the expectations across their 
courses and the ability to express in 
terms of alternative assessments how 
they have achieved/mastered the 
learning outcomes. In other words, what 
they know and are able to do as a result 
of their teaching and learning 
experiences. To alleviate participant 


36 



FIGURE 1. ATC Online Program Learning Outcome & Assessment 

Identification Process 
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concerns regarding assessing student 
learning online, step three will be 
utilized to determine best assessment 
practices for the online environment and 
for documentation of evidence. This 
documentation will guide the continuous 
improvement efforts based upon 
assessment results, step four. The fifth 
and final step within this process will be 
to reassess and revise based upon the 
evidence available and the experiences 
of the faculty with the methods selected. 

The purpose of any educational 
program is to provide quality teaching 
and learning experiences for its teachers 
and students. Making the decision to 
move pieces of the ATC Program online 
was not and still is not an easy one. New 
issues emerge everyday as instructors 
and students take this journey into the 
world of online education. These 
concerns are important and must be dealt 
with so as to not lose focus on the 
program’s purpose or quality. 

Further research is needed which 
focuses upon other methods of 
establishing learning outcomes and how 
these outcomes are assessed in similar 
online programs. The effectiveness of 
student learning within these online 
programs should also be addressed. The 
implications of receiving an education 
online are serious as it relates to teacher 
effectiveness and longevity in the 
profession. Being an online program is 
more than being seen as “competitive,” 
but rather to offer students quality 
learning opportunities with new and 
unique connections and ways of 
engagement without jeopardizing the 
ultimate goals of a teacher certification 
program. The world in which we live is 
becoming a flat world and technological 
innovations are a way of life, and to 
prepare students for this is imperative as 
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the experiences they gain from the ATC 
Program, they will pass along to their 
future students. To be or not to be 
online? To be! 
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